[Feat] Talk naturally with Fast sessions using GPT-Live - #1994
Conversation
|
No code issues found. See task
Reviewed 83805bc |
Adds a hands-free voice mode to the session composer: microphone audio streams to OpenAI realtime transcription over WebRTC (ephemeral, transcription-scoped tokens minted server-side), completed utterances send through the normal fastSessions.reply path, and replies are read aloud via streaming gpt-4o-mini-tts PCM with barge-in on detected speech. Voice rides R_VOICE_OPENAI_API_KEY with OPENAI_API_KEY fallback; the key stays on the control plane and unset keys hide the feature.
gpt-live-transcribe streams word-by-word deltas but rejects server-side turn_detection, so the browser now runs a lightweight energy-based VAD: a pause commits the audio buffer to finalize the utterance, and detected speech interrupts reply playback (with a higher threshold while audio is playing so speaker echo does not cut the agent off).
3a7767a to
e7e280f
Compare
…ce in transcript tests - Spoken replies now wait for the composite agent-working signal (send in flight, turn responding, response pending) to clear instead of the first visible assistant message, so a Fast progress kickoff no longer swallows the real result. - useLiveVoice tracks a start generation: a stop() or repeat start() during the handshake makes the stale attempt release its mic and peer instead of activating after the user cancelled. The composer toggle also cancels a connecting handshake. - FastSessionTranscript.client.test.tsx mocks voice.status and useLiveVoice, and adds coverage for the toggle and turn-settle speech.
- The spoken-reply cutoff is now the newest transcript timestamp at the moment voice starts instead of the browser clock, so a client clock running ahead of the server no longer silences every reply. - A pending structured input request replaces the composer and its voice controls, so the conversation is stopped when one arrives rather than leaving the microphone open with no End control.
The voice loop waited for the whole Fast turn to finish, then sent the entire reply to TTS as one request, so the user heard nothing until closeout plus first-byte on a long synthesis. - Speak incrementally: each completed sentence of a streaming reply is queued the moment it lands, the remainder when the persisted row finalizes it. A per-message cursor keeps the persisted row from repeating what the stream already said, and progress messages are read without ever skipping the result that follows. - Pipeline TTS: speak() now appends to a queue drained by one loop that keeps two synthesis requests in flight ahead of playback, uses short (~400 char) requests for fast first byte, and coalesces sentences that arrive while a request is pending. - Barge-in mutes the rest of the interrupted reply rather than pausing it (surfaced as an interruptions counter for the transcript). - VAD pause 800ms -> 600ms; the server memoizes the OpenAI key lookup for 30s since synthesis is now many small requests.
…omposers The home page composer and the New Session dialog get the same live voice toggle as the session composer. A session needs content to exist, so the composer listens for the first utterance, starts the Fast session with it (plus any typed text), and opens /sessions/<id>?voice=1. The session page reads that flag, starts voice as soon as the deployment confirms it is configured with no spoken cutoff so the first reply is read aloud, and drops the flag from the URL so a reload does not restart the conversation. - TaskPromptInput gains optional voice controls and a banner slot for the status strip; dictation is disabled while a conversation is active. - useVoiceEnabled shares the status lookup between the transcript and the new-session form. - Voice stays hidden for environment launches (those are tasks, not sessions) and on deployments without voice configured. - Docs updated.
|
Updated the voice implementation to GPT-Live-1 and addressed the current review findings in Focused client and server tests, type checks, lint, docs validation, and pre-push checks pass. The branch includes current |
- Stop trimming the WebRTC SDP offer in the tRPC input schema; the trailing CRLF is required and OpenAI rejected every offer with "unmarshal SDP: EOF". - Move the voice autostart flag out of the client component module so the server page reads the string instead of a client-reference stub, and drop the StrictMode-hostile "already started" guard. - Remove the Listening/End status strip and live caption; the voice button is the only control and errors surface as a toast. - Clean each spoken utterance with a small non-reasoning model before it enters the transcript, falling back to the raw text on failure. - Tell GPT-Live which repositories, environments, and integrations the Fast session can reach, and feed the same names to cleanup as vocabulary. - Play short start and stop cues; the New Session handoff ends silently. - Include OpenAI error bodies in thrown errors and log start failures.
Route transcript cleanup through the shared non-task inference path with modelRole 'small' instead of calling OpenAI directly with a hardcoded model. Cleanup now follows the operator's helper-model setting and works on deployments whose inference is not on OpenAI; the OpenAI key is only needed for the GPT-Live session itself.
…om the first utterance - Resolve voice only from R_VOICE_OPENAI_API_KEY; drop the OPENAI_API_KEY fallback so enabling OpenAI for task inference never turns voice on and never bills a GPT-Live session unintentionally. - Add a kickoff voice mode for the home page and New Session dialog: GPT-Live is told the first thing the person says is their request, delegates it at once, and stays silent. The new Session opens in voice mode and speaks the reply. - Prime the new Session's conversation with the request that created it via a session-wide instructions append, since that conversation never heard it. - Log GPT-Live errors, session-close, data-channel-close, and peer connection state transitions in the browser console.
…erance - Instruct GPT-Live to read commentary word for word, in order, without paraphrasing or adding remarks, and to delegate every utterance including greetings and short remarks instead of answering on its own. - Speak each reply sentence by sentence while it streams: completed sentences go out as soon as they exist and the persisted row finishes the tail, tracked per message so nothing is read twice. - Add a silence-flush safety net in the voice hook: speech that GPT-Live never delegates is sent to Fast after 1.5s of quiet with a null delegation, and a delegation arriving right after is treated as already handled. - Attribute streamed reply pieces to the most recent spoken request until the persisted row pins the exact turn.
Voice now follows the call model: GPT-Live is Roomote's voice, Fast is the worker, and the Session transcript is the record of what was said. - GPT-Live policy: acknowledge in a few words, delegate anything involving code or work, answer small talk itself, report results faithfully with numbers and names kept exact. No more verbatim reading or delegate-everything rules. - Spoken requests run Fast turns in voice mode (new voiceMode flag through the reply command, durable replay, and answerFastAgentQuestion). The Fast prompt gains a Voice Call section, and the reply is marked voiceCommentary: it streams to GPT-Live sentence by sentence and renders as a collapsed "Reported result to voice" row rather than a chat bubble. Typed messages during a call are still answered in writing. - The call is transcribed into the Session: what GPT-Live said (its output transcript) and what the person said when it answered directly are persisted as voice turns via voice.recordTurn, and added to Fast's history so later requests can refer to the call. Call started/ended markers are a new voice_call event type rendered as dividers with the duration. - The home page and New Session dialog open an empty Session and start the call there, replacing the kickoff Live mode and context priming; a pre-typed message is sent as the first turn. - In-call controls in the composer: timer, mute microphone, silence output.
…ies exactly - Show both sides of the call as they are spoken: GPT-Live's words and the person's words render as in-progress bubbles that fill word by word, stay while their voice-turn row is written, and hand over to the persisted row (or to the reply's optimistic row for a delegated request). - Carry the Fast turn id on reply-stream chunks (`fastTurnId`) so streamed sentences are attributed to the delegation that asked for them, even when a second spoken request arrives before the first reply's persisted row. Removes the last-delegation fallback and the responding-window heuristic. - Run a typed kickoff in voice mode when the Session is opened for a call, so the first reply is spoken; an empty voice Session schedules no turn.
GPT-Live acknowledges the instant it delegates, before the request has been cleaned up and sent, so the acknowledgement row was persisted first and sorted above the request. The hook now exposes how many utterances are between speech end and delivery, and the transcript holds spoken turns while a request is in flight, recording them once it has been sent.
GPT-Live's input transcript carries non-speech tags such as [chuckle] and [tongue click]. Strip them from what the person said before it reaches the live bubble, the recorded turn, or the Fast request; annotation-only speech is not a turn.
Inferring the in-flight state from send status and queue length left a gap between dequeue and the reply mutation. Count spoken requests from enqueue until their mutation settles and flush held acknowledgements only when that count and the cleanup count are both zero.
A [chuckle] in GPT-Live's output transcript can be cut in two when the person starts talking; each half then missed the bracket matcher and was recorded as its own spoken turn. Strip an unfinished tag at the end of a turn and a tag tail at the start of the next.
What changed
Voice is a phone call with Roomote, transcribed into the Session.
gpt-live-1for one native full-duplex WebRTC conversation with interruption handling. GPT-Live is Roomote's voice: it acknowledges each request in a few words, hands anything involving code or work to the Fast Session through client delegation, answers greetings and small talk itself, and reports results faithfully in its own words, keeping numbers, names, paths, and link labels exact.voice_calltranscript event). The Session's selected model, tools, permissions, and transcript are preserved throughout.POST /v1/live/sessions; the OpenAI API key stays server-side and the browser receives only the negotiated SDP answer. Voice is opt-in through its ownR_VOICE_OPENAI_API_KEY, with no fallback to the generalOPENAI_API_KEY. Upstream error bodies are included in server errors, and GPT-Live errors, session close, and peer connection state are logged in the browser console.R_VOICE_OPENAI_API_KEYrequirement.Why this change was made
GPT-Live provides the natural, interruption-friendly conversation layer while Roomote's existing Fast harness remains the authoritative task agent. Making the voice the author of spoken replies, and the transcript the record of the call, avoids the double answers that come from having Fast write a chat reply and GPT-Live speak at the same time, and matches how GPT-Live is designed to be used.
Impact
Users can talk to the same Fast Session they type to, hear its results, and read the call back later in the Session. Deployments need an OpenAI project with
gpt-live-1access andR_VOICE_OPENAI_API_KEYset; without it, the voice controls remain hidden.Verification
Exercised end to end against real OpenAI Live from a local deployment: home-page button opens an empty Session and the call auto-starts there with the Call started marker; mute toggles; a typed message during the call is answered in writing and not spoken; ending the call writes the Call ended marker with its duration. Earlier in the branch: session handshake, spoken delegation into Fast, kickoff and reconnection flows.
Bugs found and fixed along the way: the tRPC schema trimmed the SDP offer's required trailing CRLF (OpenAI rejected every offer with
failed to unmarshal SDP: EOF); the autostart flag constant lived in a client module and reached the server page as a client-reference stub; the autostart effect's guard did not survive React StrictMode.Lint, types, knip, and the voice hook, composer, transcript, home, server, and command test suites pass.